The Bonferonni and Šidák Corrections for Multiple Comparisons

The more tests we perform on a set of data, the more likely we are to reject the null hypothesis when it is true (i.e., a “Type I” error). This is a consequence of the logic of hypothesis testing: We reject the null hypothesis if we witness a rare event. But the larger the number of tests, the easier it is to find rare events and therefore the easier it is to make the mistake of thinking that there is an effect when there is none. This problem is called the inflation of the alpha level. In order to be protected from it, one strategy is to correct the alpha level when performing multiple tests. Making the alpha level more stringent (i.e., smaller) will create less errors, but it may also make it harder to detect real effects.